What Makes a Good Empirical Software Engineering Thesis?: Some Advice

An empirical software engineering (ESE) PhD thesis has some special features, which makes it slightly different from a thesis any other different field. One of the differences between the two is the intensive use of empirical studies in an ESE dissertation. This talk starts by giving students advice on what makes a good ESE PhD thesis in the form of a list of do’s and don’ts. The keynote later discusses what different empirical studies can be used (surveys, case studies and experiments). Finally, it focuses on one specific type of empirical study: controlled experiments. Experimentation is a risky business, and software engineering (SE) has some special features, leading to some experimentation issues being conceived of differently than in other disciplines. Some advice is given on how to analyse SE experiments. Keywords— empirical software engineering, PhD thesis. I. WHAT IS AN EMPIRICAL SOFTWARE ENGINEERING THESIS? A few years ago, PhD theses did not use to include empirical validation. However, times are changing, and, nowadays, any PhD thesis must include at least some kind of empirical validation. This does not mean, however, that any PhD thesis with an empirical component is an empirical software engineering (ESE) thesis. The key characteristic of ESE theses is that they have a major empirical component. There are different types of ESE PhD theses. There is no formal typology, but, if we look at the ESE PhD theses written over the last 20 years, two main types stand out: 1) theses gathering knowledge about a specific topic by means of empirical studies, and 2) theses proposing methodological advances in ESE. A. Theses Gathering Knowledge about a Specific Topic Examples of such theses are (in chronological order): • Seaman [11] conducts an empirical study whose goal is to characterize certain aspects of communication among members of a software development organization. • Shull [13] runs a series of experiments to develop a body of knowledge on reading techniques for inspections. • Thelin [15] reports a series of experiments on a reading technique called usage-based reading. • Carver [1] studies the impact of an inspector’s characteristics (background and experience) on his or her effectiveness in a software inspection. The thesis may, in some cases, take in methodological aspects. For example, Seaman addresses the problem of how to analyse qualitative data. Shull tackles the problem of synthesizing the results of the different studies run. In other cases, techniques that have not been used in SE before are applied. For example, Carver uses grounded theory [5]. B. Theses Proposing Methodological Advances in ESE Examples of such theses are (in chronological order): • Daly [3] proposes a multi-method approach to empirical research, which, when integrated with the technique of replication, outputs more reliable and generalizable results. • Ciolkowski [2] proposes an approach for the quantitative aggregation of evidence from controlled experiments in software engineering (SE). • Jedlitschka [6] deals with the problem of reporting the results in SE experiments so that they are useful for software managers for decision making. • Solari [14] addresses which contents a laboratory package for running SE experiments should have. • Gómez [10] proposes a taxonomy for replications in SE. The taxonomy is used as a driver to plan the order in which replications should be run and how their results should be aggregated. In all cases, empirical studies are run (or used) as a means to develop and validate the research performed. II. DO’S AND DON’TS Irrespective of the thesis type, some general guidelines can be established around three key issues that a PhD thesis should address: tackled problem, research method used and publication. A. Tackled Problem Regarding the definition of the problem: • DON ́T take it for granted that everybody is aware of Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes.